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Abstract 

Background: Funders of medical research the world over are increasingly seeking, in research assessment, to 
complement traditional output measures of scientific publications with more outcome-based indicators of societal 
and economic impact. In the United Kingdom, the Higher Education Funding Council for England (HEFCE) 
developed proposals for the Research Excellence Framework (REF) to allocate public research funding to higher 
education institutions, inter alia, on the basis of the social and economic impact of their research. In 2010, it 
conducted a pilot exercise to test these proposals and refine impact indicators and criteria. 

Methods: The impact indicators proposed in the 2010 REF impact pilot exercise are critically reviewed and 
appraised using insights from the relevant literature and empirical data collected for the University of Oxford's REF 
pilot submission in clinical medicine. The empirical data were gathered from existing administrative sources and an 
online administrative survey carried out by the university's Medical Sciences Division among 289 clinical medicine 
faculty members (48.1% response rate). 

Results: The feasibility and scope of measuring research impact in clinical medicine in a given university are 
assessed. Twenty impact indicators from seven categories proposed by HEFCE are presented; their strengths and 
limitations are discussed using insights from the relevant biomedical and research policy literature. 

Conclusions: While the 2010 pilot exercise has confirmed that the majority of the proposed indicators have some 
validity, there are significant challenges in operationalising and measuring these indicators reliably, as well as in 
comparing evidence of research impact across different cases in a standardised manner. It is suggested that the 
public funding agencies, medical research charities, universities, and the wider medical research community work 
together to develop more robust methodologies for capturing and describing impact, including more valid and 
reliable impact indicators. 



Background 

In the United Kingdom, universities and other higher 
education institutions (HEIs) conduct more than £1.3 
billion worth of research in clinical medicine annually, 
most of which is funded by United Kingdom (UK) and 
European Union (EU) government agencies as well as 
medical research charities. Owing to the support of the 
public and taxpayers, the UK has developed some of the 
strongest and most productive clinical medicine research 
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bases in the world. According to Thompson Reuters 
(ISI) bibliometric indicators, the UK's clinical sciences 
research is second only to the USA. With just 0.9% of 
the worlds population, the UK produces 8.7% of world 
publications in clinical sciences and generates 12.7% of 
world citations [1]. 

Increasingly, however, funders of medical research 
the world over are seeking, in research assessment, to 
complement traditional output measures of scientific 
publications - such as number of publications, num- 
ber of citations, impact factor, research funding, de- 
gree of co-authorship, and /z-index [2,3] - with more 
outcome-based indicators of societal and economic impact 
[4-15]. The medical research and academic community is 
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wondering how to measure returns on investment in 
health research [16,17], how to "best report to taxpayers 
and philanthropists on the societal value produced by 
the monies entrusted to [it]" [18], and how key indica- 
tors in academic medicine "may promote effective 
growth and development in a dynamic clinical, training, 
and research environment" [19]. Medical schools around 
the world are striving to achieve and demonstrate a 
greater impact on the health needs of the populations 
and societies they serve as part of their social account- 
ability strategies [20-23]. 

For medical research charities, an important rationale 
for more outcome-based evaluation of the research they 
fund is to fulfil more effectively the wishes of their bene- 
factors, e.g. "the improvement of the physical conditions 
of mankind" in the case of the Sir Henry Wellcome 
Trust [24]. For some collection-based charities, such as 
the UICs Arthritis Research Campaign, the virtue of 
demonstrating the outcomes of research they fund lies 
in that demonstrable outcomes help them compete for 
contributions [25]. Overall, medical research charities 
hope that improved understanding of how research 
funding impacts on health outcomes will enable them 
to [26]: 

• "show accountability and good research governance 
to their stakeholders; 

• enhance public perception and understanding of 
biomedical science and the scientific process; 

• and allow the development of more effective 
strategies in research and development to increase 
the likelihood of successful' research outcomes." 

For EU and UK government agencies, the agenda be- 
hind outcome measures transcends health to encompass 
innovation, economic growth, and social progress. The 
Lisbon European Council (2000) set out a strategy for 
making Europe "the most competitive and dynamic 
knowledge-based economy in the world, capable of sus- 
tainable economic growth with more and better jobs and 
greater social cohesion" [27]. In conjunction with this 
strategy, the European Commission has argued for 
"increased and more effective public expenditure [on R&D 
and innovation]" [28], and for "the development and pilot- 
ing of indicators designed to measure the social and eco- 
nomic impact of research in general, and of European/ 
international collaborative research in particular" [29]. In 
the UK, the research strategy for the National Health Ser- 
vice (NHS) "Best Research for Best Health" aims both "to 
improve the nations health and increase the nations 
wealth" [30], while the performance monitoring frame- 
work developed by the UICs National Institute for Health 
Research (NIHR) links early performance indicators with 
longer-term research impacts [31]. 



In 1989, the UK was the first country in the world to im- 
plement a performance-based research funding system, 
the Research Assessment Exercise (RAE), and since then 
at least thirteen more countries, including Australia, New 
Zealand, Hong Kong (China), and several EU countries, 
have introduced such systems [32]. The RAE was con- 
ducted approximately every five years to assess higher 
education-based research activity for the purpose of allo- 
cating core public research funds to universities and other 
HEIs. Those who performed well had their public research 
funding increased, while those who underperformed had 
it decreased. Traditionally, the focus of the RAE has been 
on "quality", seen as academic or scholarly excellence and 
assessed through the criteria of rigour, originality and sig- 
nificance, coupled with other indicators, such as academic 
esteem, viability of research environments, and research 
capacity [33]. These criteria have been interpreted in each 
subject-group (known as "unit of assessment") by dedicated 
sub-panels of assessors, who peer-reviewed the different 
parts of institutional submissions and used a common 
grading scale to arrive at quantified quality profiles. 

In 2014, the RAE will be replaced by a new performance- 
based research funding system - the Research Excellence 
Framework (REF). The aims of the REF are similar to the 
RAE, i.e. primarily to provide a basis for resource alloca- 
tion, accountability for public investment in research, and 
benchmarking information and reputational yardsticks for 
the higher education sector [34]. A major new development 
in the REF was the decision to base the future allocation of 
public research funding to universities and other HEIs on, 
inter alia, the social and economic impact of research [35]. 
The introduction of impact assessment was wrought with 
controversy as it was perceived by many, especially in social 
sciences and humanities, as "a threat to researchers' auton- 
omy and to fundamental academic freedoms" [36]. Given 
that universities in the UK are independent from the gov- 
ernment and that academic professions are self-regulated, 
elected politicians in the government cannot hold univer- 
sities directly accountable for the type of research they 
choose to conduct. Therefore, in order to achieve its goal 
of increasing the economic and social impact of publicly- 
funded research within specified time horizons, the govern- 
ment introduced funding incentives for universities to en- 
gage in high-impact research. This decision was viewed by 
many as the governments attempt to limit universities' 
freedom to pursue all forms of research, including research 
that may not necessarily lead to high impact, and to hold 
independent academic professions accountable for some- 
thing that they had not chosen themselves [36]. 

In academic clinical medicine, the introduction of im- 
pact assessment was perceived as less controversial. Due 
to their focus on translational research and health out- 
comes, academic physicians and scientists were used to 
outcome-based indicators, and there had been growing 
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global consensus among medical schools about the need 
to strengthen their social accountability [22]. The latter is 
defined by the World Health Organization as "the obliga- 
tion to direct their [medical schools'] education, research 
and service activities towards addressing the priority 
health concerns of the community, region, and/or nation 
they have the mandate to serve. The priority health con- 
cerns are to be identified jointly by governments, health 
care organisations, health professionals and the public" 
[20]. Importantly, the notion of social accountability im- 
plies that it is measured and reported back to society [23]. 

The proposals to include indicators of social and eco- 
nomic impact in the UK's performance-based research 
funding system were first tested in a pilot exercise run 
by the Higher Education Funding Council for England 
(HEFCE) on behalf of four UK higher education funding 
bodies in 2010. The impact pilot exercise aimed to de- 
velop a coherent approach to assessing impact in the 
REF. It invited participating institutions to submit details 
of economic and societal impacts in several disciplines 
(including clinical medicine), in the form of a) an impact 
statement, and b) case studies of specific impacts, pro- 
duced using a generic impact template. The impact 
statement was designed to provide "evidence of the 
breadth of the units contributions to society or the 
economy" [37]. It had to include any appropriate indica- 
tors of impact, collaboration, and other interactions with 
research users (such as the NHS, UK and EU govern- 
ment, home and overseas industry, charities, and re- 
gional development agencies), which were deemed 
applicable to the whole subject-group ("unit of assess- 
ment"). HEFCE intended to use this information to- 
gether with research income from key research users to 
explore the feasibility of developing standardised impact 
indicators. The case studies were designed to illustrate 
the units contributions in more detail. Although partici- 
pating institutions could choose which impacts to in- 
clude, they were encouraged to submit case studies 
illustrating a wide range of impacts. For every ten faculty 
members, participating institutions were asked to submit 
one case study. The generic impact template for each 
case study included information on 1) the nature and ex- 
tent of a specific impact, including appropriate indica- 
tors; 2) how the units research activity contributed to 
this impact; and 3) references to external sources that 
could corroborate the information about the impact and 
its underpinning research [37]. 

For the purpose of the 2010 pilot exercise, HEFCE 
defined research impact as "any identifiable benefit 
to, or positive influence on, the economy, society, 
public policy or services, culture, the environment or 
quality of life," and provided HEIs with a "common 
menu" of impact indicators in the following broad 
categories [37]: 



• Delivering highly skilled people; 

• Creating new businesses, improving the 
performance of existing businesses, or 
commercialising new products or processes; 

• Attracting R&D investment from global business; 

• Better informed public policy-making or improved 
public services; 

• Improved patient care or health outcomes; 

• Progress towards sustainable development, including 
environmental sustainability; 

• Cultural enrichment, including improved public 
engagement with science and research; 

• Improved social welfare, social cohesion or national 
security; 

• Other quality of life benefits. 

On the basis of the evidence gathered through the 
2010 pilot exercise [38] and a public consultation [39], 
the funding bodies decided to invite HEIs to submit, in 
the next round of research assessment under the REF in 
2014, a statement about their approach to, and strategy 
for, enabling research impact, together with a set of im- 
pact case studies containing "a narrative that includes 
indicators and evidence as appropriate to the case being 
made" [35]. The definition of impact was more clearly 
focused on non-academic effects, changes or benefits 
[34]. It was also decided that the assessment of impact 
will account for 20% of the overall assessment outcomes, 
alongside the quality of research outputs (65%) and the 
vitality of the research environment (15%) [35]. The 
funding bodies set out the generic assessment criteria of 
"reach" (or breadth) and "significance" (or intensity) of 
impact, while the discipline-specific interpretations of 
these criteria were left to the panels and sub-panels re- 
sponsible for the actual assessment [40]. The "common 
menu" of indicators was dropped from the generic guid- 
ance, but re-worked forms of it were maintained in the 
statements of criteria and working methods of the four 
main panels of assessors, including Panel A, of which 
clinical medicine is a part [40]. 

As indicated by the rich body of research on the influ- 
ence on HEIs of the previous rounds of the RAE, the 
newly- introduced impact assessment for resource alloca- 
tion purposes via the REF is likely to affect strategic 
decisions in HEIs at institutional and departmental level, 
as well as the individual behaviour of researchers and re- 
search teams [41-45]. Given the aims of the REF and the 
weighting of 20% given to impact in the overall outcome, 
HEIs in the UK are already keenly aware of the higher 
stakes involved. Particularly challenging is the fact that 
the development of impact assessment methodologies 
has still to address major difficulties, such as that of 
causality, of operationalisation, of attribution, of track- 
ing long-term impacts, of combining quantitative and 
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qualitative indicators, or of balancing the concern for 
sensitivity to field and context specificities with that for 
comparability across cases [12,36,46-51]. In addition, 
prior to the REF, institutions and individual researchers 
had not been expected to closely monitor and account 
for impact-related activities; thus, at the start of the 
current REF assessment period, impact records, if any, 
were at best patchy, and fit-for-purpose information and 
management systems were not available. Some competi- 
tively allocated project funding streams (such as those by 
the Research Councils UK, including the Medical Re- 
search Council) introduced explicit requirements to 
report on engagement, dissemination or knowledge ex- 
change activities. However, as the overlap between these 
activities and the REF definition of impact was only 
partial, the systems that had been put in place for this 
purpose sometimes added to the confusion, rather than 
making it easier, about reporting on impact for REF 
purposes. As such, the forthcoming REF is facing ser- 
ious difficulties in simultaneously attempting to concep- 
tualise, operationalise, and assess impact. 

In this article, we critically review and appraise the im- 
pact indicators used by HEFCE to inform the 2010 im- 
pact pilot exercise, using insights from the relevant 
literature and empirical data gathered for the University 
of Oxfords submission to the pilot exercise in clinical 
medicine. We show that, while the majority of the pro- 
posed indicators have some validity, there are significant 
challenges in operationalising and measuring these indi- 
cators reliably, as well as in comparing evidence of re- 
search impact across different cases in a standardised 
and consistent manner. We argue that this unevenness 
needs to be addressed early on in the design of method- 
ologies aimed at capturing and assessing impact, and 
that more work, involving collaboration among research, 
beneficiary, and funding partners, is required to ensure a 
research-informed approach to impact assessment in the 
UK REF and beyond. 

In what follows, we begin with a methods section 
explaining why clinical medicine at the University of 
Oxford provides a critical opportunity to study impact 
assessment, and describing the design of the 2010 im- 
pact pilot exercise. We then present the empirical data 
and discuss the strengths and limitations of the pro- 
posed impact indicators using insights from the relevant 
biomedical and research policy literature. Finally, we draw 
conclusions and make recommendations for the develop- 
ment and validation of impact indicators. 

Methods 

A total of 29 HEIs, including the University of Oxford, 
participated in the 2010 impact pilot exercise in one or 
more subject-groups ("units of assessment"): clinical 
medicine, physics, earth systems and environmental 



sciences, social work and social policy & administration, 
and English language & literature [37]. Clinical medicine 
is the largest subject-group, by research income, in the 
UK, and as such it provides a critical opportunity for the 
development and validation of impact indicators. Out of 
£4,145 million worth of research grants and contracts 
awarded to UK HEIs in 2008/09 (i.e. at the end of the 
census period for the pilot exercise), clinical medicine 
awards accounted for £1,347 million across all HEIs, 
including £174 million awarded to the University of 
Oxford alone (Figure 1) [52]. 

This article uses administrative data from the University 
of Oxford to operationalise the "common menu" of impact 
indicators proposed by HEFCE and to test their relevance 
to research practice in clinical medicine. In 2010, in prep- 
aration for the pilot exercise, the university's Medical 
Sciences Division collected a first round of relevant ad- 
ministrative data from university sources. In addition, the 
Division developed an online administrative survey, based 
on the HEFCE "common menu" of impact indicators; the 
survey was conducted among the faculty members form- 
ing the university's clinical medicine "unit of assessment". 
In 2011, a second round of data was gathered by the 
authors from the relevant administrative units of the uni- 
versity and publicly available sources specifically for the 
purpose of the current study including updated data on 
commercialisation activities, clinical trials, and financial 
indicators. The university's Clinical Trials and Research 
Governance Team reviewed the study and deemed neces- 
sary no further ethics committee clearance. Institutional 
approval for the use of the administrative data used in the 
paper was secured from the Medical Sciences Division. 

In March 2010, instructions and an electronic link to 
the impact assessment survey were e-mailed to the 289 
faculty members who were most likely to participate in 
the 2014 REF, i.e. those who participated in the 2008 RAE 
in the "units of assessment" relevant to clinical medicine 
and who at the moment of the survey had active university 
e-mail accounts. The survey included 15 open-ended 
questions structured around the proposed impact indica- 
tors, e.g. "If in the period between January 2005 and 
December 2009, you participated on public policy/advis- 
ory committees, please indicate how many times and spe- 
cify their topics and your capacity". The survey also 
included a personal details section and three questions 
prompting respondents to provide open-ended comments 
or thoughts on the impact indicators, on the impact pilot 
exercise, and on how the Medical Sciences Division should 
organise the collection of impact data on a regular basis in 
the least burdensome way for faculty. 

The survey received a total of 139 responses, including 
four responses after the closing date, which all were 
accepted for analysis here (48.1% response rate). The 
achieved response rate was relatively good for voluntary 
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Figure 1 Top 10 UK HEIs by income from research grants and contracts in clinical medicine, 2004/05-2008/09. Source: Higher Education 
Information Database for Institutions (heidi), 2010. 



online surveys involving physicians both in the current re- 
search setting [53] and elsewhere [54] . Sample proportions 
by gender, career stage, and staff category were similar to 
the population surveyed; however, women, early-career 
researchers, and mid-career researchers were slightly over- 
represented, whilst senior researchers were slightly under- 
represented; there were also slight discrepancies within 
staff categories (Table 1). Staff grades from the university's 
payroll were used to determine career stage and staff cat- 
egory, and these were unknown for 5 researchers (1.7%), 
who were no longer on the payroll. Overall, the survey 
provided a unique and adequate insight into individual 
and aggregated evidence of impacts and into clinical medi- 
cine faculty members' views on the nature and appropri- 
ateness of the impact indicators proposed by HEFCE. 



Table 1 Demographic and professional characteristics of 
clinical medicine faculty, 2010 



Characteristic 


Population 

(N=289) 


Sample 
(n=139) 


Gender 


Female 


21.5% 


22.3% 


Male 


78.5% 


77.7% 


Career stage 


Senior (professorial) 


43.3% 


41.0% 


Mid-career 


43.9% 


46.0% 


Early-career 


11.1% 


12.9% 


Unknown 


1 .7% 


0.0% 


Staff category 


Category A, university scientists 


71.3% 


75.5% 


Category A, university clinical academics 


18.7% 


18.0% 


Category C, MRC scientists and NHS clinicians 


8.3% 


6.5% 


Unknown 


1 .7% 


0.0% 



Results and discussion 

In the main part of this paper, we assess the scope and 
relevance to clinical medicine research of a range of mea- 
sures that operationalise the impact indicators proposed. 
We unpack each of the potential indicators listed above, 
track its rise as an aspect of research deemed important 
by a range of interested parties, discuss some of the mea- 
sures most commonly used to assess it, and illustrate the 
strengths and limitations of these measures using data 
from clinical medicine in Oxford. Data are presented in 
anonymised and aggregated format and relate to the five- 
year census period January 2005 - December 2009, unless 
otherwise stated. 

Delivering highly skilled people 

Staff movement between academia and industry 

Increasing human mobility between academia and in- 
dustry is a desired goal not only in the UK, but also in 
EU and OECD countries - it is perceived to enhance 
knowledge transfer in both directions and to offer better 
employability and career prospects for researchers [55- 
57]. The Lambert Review of business-university collab- 
oration in the UK concluded that "the best forms of 
knowledge transfer involve human interaction" [58]. 
Moreover, research evidence from the US suggests that 
inter-sectoral collaborations and staff movements from 
academic to industrial jobs and vice versa are positively 
associated with researchers' productivity, most likely, be- 
cause of the accumulation of scientific and technical 
human capital in multiple settings [59,60]. There is also 
some normative guidance on the desirable rate of staff 
movement. Following the 2005 summit of EU leaders in 
Hampton Court, the Aho Report on creating an innova- 
tive Europe suggested that "[t]en per cent of the work- 
force in each year should be moving" [61]. 
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Although our analysis of administrative data from 
Oxford showed minimal full-time movement of tenured 
researchers between the sectors, a significant number of 
university researchers worked in industry part-time, on 
temporary assignments, or in their own time; conversely, 
many industry researchers held visiting positions in the 
university. More than one-third of respondents to the 
impact assessment survey (37%) reported spending time 
working in, or providing advice and consultancy to, in- 
dustry, through ad hoc research projects and collabora- 
tions, long-term industrial-academic partnerships, or 
permanent appointments in companies' scientific and ad- 
visory committees and non-executive boards. The range 
of companies included global biotechnology and pharma- 
ceutical companies, small and medium enterprises, and 
university spin-outs. The amount of time spent in industry 
by each respondent ranged from several working days or 
weekends to two months per annum. 

These data suggest that there is considerable industry 
demand for the expertise of university researchers, but 
this demand is not (yet) accompanied by the full-time 
job changes characteristic of the US-style "revolving 
door" model of academia-industry employment [62]. 
Similarly, according to the 2011 Careers in Research On- 
line Survey (CROS) of research staff in UK HEIs, the 
proportion of staff reporting industry and business pla- 
cements and secondments across the HE sector is still 
low (5% and 7%, respectively), while other types of colla- 
borations with industry are more prevalent (36% of the 
total respondents) [63]. Consequently, in documenting 
and assessing this area of impact, the absolute num- 
ber and proportion of academic and research staff 
who worked part-time in, or provided advice and 
consultancy to, industry, their roles, and the time 
spent working in or with industry need to be taken into 
account. 

Employment of post-doctoral researchers in industry 

The Roberts Review of the supply of people with science, 
technology, engineering and mathematics skills in the 
UK acknowledged that post-doctoral and other contract 
research staff (CRS) offer universities project-based 
skills, staffing flexibility, new innovative approaches, and 
other advantages [64]. Nonetheless, limited permanent 
academic employment opportunities, not only in the 
UK, but also internationally, cause concerns that CRS 
may spend a long time moving from one temporary pos- 
ition to another, to the detriment of their professional 
development, career prospects, and quality of work and 
life conditions [65]. For example, 14% of the research 
staff surveyed in CROS 2011 reported having held five 
or more different contracts of employment as research- 
ers with their current institutions [63]. A further con- 
cern arising from this situation is that highly-skilled 



graduates would be less interested in pursuing research 
careers [66-69]. Roberts argued against CRS remaining 
on a series of short-term contracts for a long period of 
time, and suggested improving CRS training in the skills 
required either in an academic or industrial career and 
that "in time. . . [the industrial career trajectory] should 
come to be regarded as the 'default option' by CRS" 
[64]. Therefore, whilst bearing in mind that industry de- 
mand for higher degree graduates varies widely across 
sectors and organisations [62], the employment of post- 
doctoral researchers in permanent positions in industry 
may be used as an indicator of the ability of universities 
to provide CRS with skills and career development 
opportunities (e.g. industrial placements and second- 
ments) to pursue a corporate research career. 

In trying to operationalise this indicator in Oxford, we 
found no clear definition for post-doctoral research posi- 
tions in the current human resources (HR) practices, 
which seems to be a common limitation of the higher 
education HR practices in many countries [68]. For this 
reason, we examined data declared on the leavers form 
to gather information on the destinations of leaving CRS 
on grades 7 and 8 as a proxy for leaving post-doctoral 
researchers. During the five-year census period, 1013 
such CRS left clinical departments; the known destina- 
tions of 762 CRS, who declared going into regular em- 
ployment or study, are as follows: 

• 57% - education and research institutions in the UK 
and overseas, 

• 22% - health services in the UK and overseas, 

• 9% - private industry/commerce or self-employed in 
the UK, 

• 12% - other employment in the UK and overseas. 

Another important limitation is the absence of any infor- 
mation on whether the subsequent jobs of CRS are per- 
manent or short-term. If a move into industry is 
accompanied by a series of short-term contracts, then this 
could be interpreted as a failure of the university in ques- 
tion to help CRS develop skills for a more permanent car- 
eer; vice versa, a move to another education or research 
institution on a permanent contract could be interpreted 
as a success. Given the lack of reliable data, developing a 
clear, applicable measure of the employment of post- 
doctoral researchers in industry seems problematic. 

Concomitantly, these data draw attention to the move- 
ment of physician-scientists and other highly- skilled 
individuals who are trained to perform translational re- 
search between universities and health services. Bridging 
research and clinical practice by increasing the pool of 
translational investigators is high on the agenda on both 
sides of the Atlantic [70-72]. In the UK, for example, the 
NIHR has established Academic Clinical Fellowships, 
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Clinical Lectureships, In Practice Fellowships, and Clin- 
ician Scientist Awards to promote integrated academic 
and clinical careers at the pre-doctoral, doctoral, and 
post-doctoral levels [73]. The efforts of universities and 
their partner teaching hospitals to increase the pool of 
translational investigators can, thus, be assessed by 
measuring the number of such awards. 

Creating new businesses, improving the performance 
of existing businesses, or commercialising new products 
or processes 

Research contracts and income from industry 

Whilst different types of research can have impact on the 
economy, the Cooksey Review of UK health research 
funding concluded that more opportunities to create add- 
itional health and wealth benefits from research lie in 
translational and applied research [74]. The review recom- 
mended that, while sustaining the current funding levels 
for basic science, "future increases in funding should be 
weighted towards translational and applied research until 
a more balanced portfolio is achieved" [74]. Empirical 
studies suggest that scientists with industry funding con- 
duct more applied research and, contrary to a popular 
assumption that industry funding negatively affects trad- 
itional scientific outputs such as publications, they also 
produce more scientific publications than scientists with- 
out industry funding [75,76]. Therefore, research contracts 
and income from industry can be used as an indicator of 
potential health and wealth benefits for the economy, and 
an increased level of industry funding may also positively 
affect the traditional scientific outputs. 

According to the Higher Education Statistics Agency 
(HESA), income from research grants and contracts in 
clinical medicine from industry in 2008/09 was higher in 
Oxford (£20.3 million) than in any other UK HEI except 
Imperial College London (£26.1 million); it accounted 
for 11.7% and 10.3% of external research income in 
Oxford and all UK HEIs respectively (Table 2) [52]. 
While research income from industry is a useful indica- 
tor, it does not allow assessment of the potential health 
and wealth benefits of the overwhelming majority of 
clinical medicine research in UK HEIs, which is funded 
by the public either through government agencies or 
charities. Yet, the pharmaceutical industry is highly 
dependent on publicly-funded research, in particular 
basic research [77]. US studies found that 31% of drugs 
and medical products could not have been developed 
(without substantial delay) in the absence of recent aca- 
demic research [78], and that 79.1% of the papers cited by 
US industry drug and medicine patents were outputs of 
publicly-funded academic science [79]. Most citations in 
patents are journal references and may be used to develop 
robust bibliometric indicators [80]. Similar to the use of 
bibliometric analyses of highly-cited publications by the 



Table 2 Income from research grants and contracts in 
clinical medicine by sponsor type, 2008/09 



Sponsor type 


University of 
Oxford 


All UK HEIs 




£M 


% 


£M 


% 


Government 


53.3 


30.7% 


558.9 


41.5% 


UK Research Councils, Royal Society 
& British Academy 


25.4 


14.6% 


222.5 


16.5% 


UK central government bodies/ local 
authorities, health & hospital authorities 


19.7 


11.3% 


282.0 


20.9% 


EU government bodies 


8.2 


4.8% 


54.4 


4.0% 


Charity 


89.0 


51.2% 


595.4 


44.2% 


UK-based charities 

(open competitive process) 


77.6 


44.7% 


454.7 


33.8% 


UK-based charities (other) 


2.4 


1.4% 


78.3 


5.8% 


PI Lh^cpH rharitipc 

(open competitive process) 


0.2 


0.1% 


3.0 


0.2% 


Non-EU-based charities 
(open competitive process) 


8.7 


5.0% 


59.4 


4.4% 


Industry 


20.3 


1 1 .7% 


138.1 


1 0.3% 


UK industry, commerce & public 
corporations 


4.3 


2.5% 


84.0 


6.2% 


EU industry, commerce & public 
corporations 


1.2 


0.7% 


14.8 


1.1% 


Non-EU industry, commerce & public 
corporations 


14.8 


8.5% 


39.4 


2.9% 


Other 


11.0 


6.3% 


54.4 


4.0% 


EU other 


1.4 


0.8% 


5.9 


0.4% 


Non-EU other 


9.6 


5.5% 


38.6 


2.9% 


Other sources 


0.0 


0.0% 


9.9 


0.7% 


Total 


173.6 


100.0% 


1,346.9 


100.0% 



Source: Higher Education Information Database for Institutions (heidi), 2010. 



NIHR to support the procurement of Biomedical Research 
Centres in England [81], HEFCE and other research flin- 
ders can use bibliometric analysis of citations in patents to 
assess the contribution of publicly-funded clinical medicine 
research to innovations in industry. 

Collaborative research with industry measured through 
co-authored outputs 

Measuring co-authored publications is a well-grounded, 
although not comprehensive, way of assessing collabora- 
tive research with industry. There is evidence from the 
US that university- industry collaborations tend to be 
driven by industry's agendas and that the resulting co- 
authored publications are less basic and more applied 
than universities would produce otherwise [82]. More- 
over, empirical studies from different countries suggest 
that co-authored publications are positively associated 
with researchers' productivity in terms of the number of 
publications and citations, but there is no conclusive evi- 
dence on whether co-authored publications are more 
likely to appear in journals with higher or lower impact 
factors than university-only papers [82-84]. However, 
there is evidence to the effect that measuring co- 
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authored publications is not a comprehensive way of 
assessing university-industry collaboration. For example, 
in the case of a Swedish medical university both industry 
funding and co-authorship indicators were shown to 
provide incomplete results [85], and a study of multi- 
disciplinary research teams in Germany concluded that 
co-authorships could account for about half of the actual 
collaborations [86]. On the whole, analysing co-authored 
publications can be a useful, but by no means exhaust- 
ive, way of assessing university-industry collaboration. 

In Oxford, 30% of respondents to the impact assess- 
ment survey reported co-authoring one or more publica- 
tions with colleagues from industry over the five-year 
census period (averaging 3.5 publications per respond- 
ent). A number of respondents also mentioned industry- 
funded publications with only academic authors, and 
further publications acknowledging industry funding. 
Furthermore, our analysis of administrative records 
revealed collaborative relationships with industry that 
were not necessarily accompanied by industry funding 
or co-authored publications, but could be accounted for 
by less conventional indicators such as memoranda of 
understanding, confidentiality agreements, collaboration 
agreements, data/material transfer agreements, and vis- 
itor agreements. Hence, it can be suggested that impact 
narratives should include additional indicators, such as 
those mentioned above, in order to capture a fuller 
range of collaborative relationships with industry. 

Income from intellectual property 

The large-scale exploitation of intellectual property by uni- 
versities is relatively new to the UK. The right of first re- 
fusal on the commercialisation of publicly-funded research, 
held for many decades by the British Technology Group, 
was only rescinded in 1985. As a consequence, universities 
gained direct control over, and exploitation of, their intel- 
lectual property, with a view to generating income for 
themselves and contributing to national wealth creation 
[87]. Not with standing conflicting views on whether com- 
mercialisation can harm or promote basic research and 
education [88], encouraging and providing universities with 
funding to engage more actively in the exploitation of intel- 
lectual property has consistently been the governments 
policy [58,89-91]. A 2009 HEFCE report concluded that 
there was strong support for knowledge exchange in HEIs 
and that it was increasingly seen by HEIs as complemen- 
tary to their traditional research and education activities 
[92]. However, the report stated that revenues from intel- 
lectual property represented a very small proportion of in- 
come from knowledge exchange [92]. Even in the US, with 
its long history of university technology transfer, revenues 
from intellectual property are rather small compared to re- 
search expenditure - most universities break even or make 
only a small amount from their investment in technology 



transfer activities [93]. Moreover, the total revenue from 
technology transfer in US universities and research hospi- 
tals is "dominated by a few very large royalties from fewer 
than 1% of total patents" and is uneven over time: 
"pharmaceutical royalties are high - but very rare," and 
"equity cash-ins from spin-outs are only occasionally large, 
and are one-time" [93]. 

Oxford academics were already entrepreneurial in the 
1950s, but it was not until the establishment of the univer- 
sity's technology transfer office in 1988, Isis Innovation, 
that Oxford became "arguably the UK's most entrepre- 
neurial university" [94]. Isis Innovation helps academics 
and researchers commercialise their work through patent- 
ing and licensing, material sales, spin-out companies, and 
consulting. Between 2004/05 and 2008/09, the total pro- 
ject income of Isis Innovation was £9.8 million, 50% of 
which may be attributed to clinical medicine. Precise attri- 
bution is somewhat problematic, because many deals were 
multidisciplinary (involving clinical, pre-clinical, and bio- 
engineering departments) and the current information 
system does not allow accurate dissection of the existing 
data in relation to academic departments. Given that in 
this five-year period Oxford's clinical medicine research 
expenditure amounted to £612.9 million [52], intellectual 
property return on research expenditure was approxi- 
mately 0.8%. Overall, the total income from intellectual 
property is a useful indicator, but currently it is problem- 
atic to attribute it accurately between multiple depart- 
ments and it represents only a small percentage compared 
to research expenditure. It seems more expedient to assess 
it on the level of the HEI as a whole and in individual 
subject-groups use case studies only of the most successful 
commercialisation activities. 

Success measures for spin-out companies 

Technology transfer through spin-out companies has 
advantages over licensing when the nature of new technol- 
ogy may not be easily patented and when universities seek 
a greater return on their intellectual property in the long 
run [95]. For all that, spinning out university companies is 
rather resource-intensive, in terms of both funding and the 
inventor's time. The Lambert Review pointed out that, due 
to the ready availability of funding for high-tech start-ups 
and an undue emphasis on spin-outs as a source of em- 
ployment, too many university spin-outs were being cre- 
ated in the UK compared to North America, including 
some of low quality [58]. Lambert suggested shifting the 
balance of commercialisation activities towards licensing 
and concentrating on high-quality spin-outs, as measured 
by their ability to attract external private equity [58]. In 
light of this, together with growth in revenue or numbers 
of employees, as proposed by HEFCE, external private 
equity backing can be taken as an important success meas- 
ure for spin-out companies. 
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In Oxford, Isis Innovation spins out more biomedical 
companies and attracts more investment than any other 
UK university [96]. In doing so, it tries to replicate the 
success of Powderject Pharmaceuticals - a vaccine, drug 
and diagnostics delivery company spun out from the 
university in 1993 and acquired ten years later by Chiron 
Corporation for $800 million [97]. During the census 
period, Isis Innovation supported the creation of 25 
spin-out companies, of which 8 related to clinical medi- 
cine in the following areas [98]: 

• Celleron - developing targeted cancer medicines; 

• TAS - developing ketone bodies as medical foods to 
increase physical and mental performance; 

• Particle Therapeutics- needleless delivery of 
therapeutic molecules across the skin; 

• Cytox- pre-symptomatic diagnosis of Alzheimer's 
Disease; 

• Clinox- provider of phase I/II trials in oncology; 

• Oxford BioDynamics- detection of aberrant gene 
expression (prognosis and diagnosis); 

• Oxford-Emergent TB Consortium -joint venture 
with Emergent BioSolutions to develop tuberculosis 
vaccine candidate, MVA85A; 

• Organox - portable device to preserve livers for 
transplantation for up to 3 days. 

During the census period, these spin-out companies 
employed a small number of people (on average 18 people 
per annum) and generated no income from sales. Never- 
theless, they raised a total of £15.7 million in external pri- 
vate equity, indicating investor confidence in future 
revenue growth. They also spent a significant proportion 
of their expenditure on R&D in universities or technology 
consultancy companies, contributing to job creation in 
these sectors. As one respondent to the impact assessment 
survey argued, the success of spin-out companies is not 
measurable over a five-year period: "our 2 spin out com- 
panies took, in one case 7 years, and in the other more 
than 10 years, before they [could] realise their full value by 
either bringing products, such as new drugs, to market or 
by being successfully sold or by going public". It can, thus, 
be argued that the success of spin-out companies should 
be evaluated over a longer period of time and that quality 
measures, such as the ability to attract external private in- 
vestment, should also be taken into account. 

Patents granted/licences awarded and brought to market 

Patenting and patent licensing is not as resource- 
intensive as spinning out companies, but seeking to in- 
crease the levels of patenting and licensing may lead to 
unintended consequences. For example, Henderson 
et al argued that the Bayh-Dole Act, which incentivised 
US universities to patent, resulted in the decline of the 



quality of university patents, as measured by citations in 
subsequent patents [99,100]. Others demonstrated that in- 
creasing the number of inventions, for which patent appli- 
cations were made and licences sought, reduced the 
average "yield" of these commercialisation activities [101]. 
Also, there are concerns that excessive proliferation of in- 
tellectual property rights may deter innovation [102], and 
that misconceptions in the value of intellectual property 
may inhibit collaboration with industry [103]. Finally, be- 
yond a certain level, higher levels of patenting are nega- 
tively associated with academic productivity in terms of 
publications [104] and their "basicness" [105]. For these 
reasons, patent numbers may be noisy indicators of know- 
ledge transfer and commercialisation. In order to minim- 
ise the negative unintended consequences of excessive 
patenting, it is important to take into account the quality 
of patents, which can be measured by licences or by cita- 
tions in subsequent patents. Licences are an appealing 
measure of quality because they indicate the economic 
value of patents and they are enforced by the competing 
interests of licensors and licensees [106]. Citations in sub- 
sequent patents are indicative of the technological import- 
ance of the antecedent patent, i.e. they draw on the 
knowledge embodied in the antecedent patent, and/or 
they may indicate that the antecedent patent had opened 
up a new field of inventive activity [100]. 

During the financial years 2004/05 to 2008/09, Isis 
Innovation filed a total of 293 new patent applications from 
university researchers in Oxford, of which 63 were based 
on disclosures from clinical medicine departments. Within 
the same period 276 licence deals were completed, includ- 
ing 61 that were related to projects derived from clinical 
medicine disclosures. There are no comparative data for 
patents and licences in clinical medicine, but in life sciences 
in general Oxford publishes more patents than any other 
university in the UK [96]; and overall Isis Innovation files 
more international patent applications to the World Intel- 
lectual Property Organization under the Patent Cooper- 
ation Treaty than any other university in Europe [107]. 
Given that there is a significant variability between UK uni- 
versities in research power and commercialisation activity, 
it is useful to take into account universities' research power 
when comparing the absolute number of patents or 
licences [96]. Moreover, given that the absolute number of 
patents does not provide information about their quality, it 
would be more effective using this measure in combination 
with additional measures, such as income from licencing 
and citations in subsequent patents. 

Attracting R&D investment from global business 
Research income from overseas business 

The globalisation of R&D has grown substantially over 
the past decades [108] to the effect that increasingly 
R&D is being outsourced to Asia, particularly China and 
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India. This promotes technological change and diffusion 
of technological advances worldwide, but raises concerns 
about the future of the domestic knowledge base in 
OECD countries and the competitiveness of their econ- 
omies [109]. In addressing such concerns, the UK's na- 
tional health research strategy, Best Research for Best 
Health, aims "to make the UK the best place in the 
world for health research, development and innovation" 
[30]. Empirical research has found a positive association 
between the productivity of academic research in host 
countries (as measured by the number of publications) 
and the level of foreign R&D investment, and that aca- 
demically stronger countries attract companies with a 
stronger science orientation in their R&D activities 
[110]. It follows that research income from overseas 
companies may be indicative of the contribution of a 
university's research to both the competitiveness of the 
domestic economy and global economic growth. 

In 2008/09, UK HEIs attracted £54.2 million of re- 
search income from EU and other overseas industry, of 
which Oxford accounted for £16 million (Table 2). 
While the percentage of non-EU overseas industry fund- 
ing in Oxford is more than twice as high as in all UK 
HEIs combined, the percentage of EU industry funding 
in Oxford is lower (Figure 2). Overall, Oxford attracts 
significantly more R&D investment from global business 
than the average UK HEI, as 78.9% of Oxfords industry 
research income comes from overseas industry. 

Better-informed public policy-making or improved public 
services 

Changes to legislation/regulations/government policy 

It has been argued that "the use of research knowledge 
to inform decision-making, not a change in health sta- 
tus, constitutes the most important generic measure of 
the impact of research that can be assessed routinely" 




University of All UK HEIs 
Oxford 



Figure 2 Income from research grants and contracts in clinical 
medicine from industry, 2008/09. Source: Higher Education 
Information Database for Institutions (heidi), 2010. 

V / 



[111]. Yet, the attribution of policy changes to particular 
research projects or funding is a methodologically- 
challenging and research-intensive task [112]. A recent 
review of research impact on policy concluded that "the 
interests of various stakeholders such as politicians, pub- 
lic servants, religious groups, pharmaceutical and diag- 
nostic companies, and health professionals may often 
run counter to the introduction of new research find- 
ings, thus, affecting policy making, budgeting, and im- 
plementation" [113]. Moreover, given the collective 
nature of decision-making in democratic societies, it is 
challenging to attribute changes in legislation, regula- 
tion, or government policy to certain teams or indivi- 
duals. As Weiss argued as early as 1979, "it probably 
takes an extraordinary concatenation of circumstances 
for research to influence policy directly" [114]. It follows 
that, for the most part, the assessment of research im- 
pact on policy requires the triangulation of data from a 
number of stakeholders using qualitative methods and a 
case study approach [113,115]. 

In Oxford, 9% of respondents to the impact assessment 
survey reported influencing changes to legislation, regula- 
tions, or government policy on the national, European, or 
international levels. The range of their activities mainly 
concerned the provision of thought leadership and advice 
to various governmental, professional, and advisory bodies. 
For example, one respondent mentioned a secondment to 
the Department of Health as a national clinical director to 
write and implement government policy for a major clin- 
ical service. In another example, a respondent mentioned 
direct contacts with the World Health Organization, in 
his/her role as president of an international professional 
society, leading to updates to guidelines for a complex area 
of practice and research. Other respondents also men- 
tioned instances of their papers being used as evidence 
during public policy deliberations, or themselves par- 
ticipating in policy advocacy campaigns. It is impor- 
tant to note that several respondents were sceptical 
about the feasibility of attributing policy changes to 
ones research-based activity, and one respondent ad- 
mitted to not knowing how to prove that his/her policy 
advocacy had contributed to policy change. A further limi- 
tation arises from the fact that assessing only documented 
influence on changes to legislation and regulation (e.g., on 
the basis of references made to research outputs in policy 
documents) will overlook the more complex, qualitative 
aspects of academics' participation in the democratic de- 
liberation and implementation of public policies. Taking 
all these into account, it seems to be more inclusive and 
less burdensome to assess researchers' participation in 
public policy processes, for example, through provision of 
advice and leadership, rather than attempting to separate 
out the contribution of specific research in achieving 
changes to legislation and regulation. 
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Participation on public policy advisory committees 

Universities can play an important role in ensuring that 
government policies "are forward-looking and shaped by 
the evidence rather than a response to short-term pres- 
sures" [116]. University academics generate an evidence 
base for public policy [117] and can facilitate the passage 
of new research knowledge into the evidence base of 
concrete policies. Their participation on public policy/ 
advisory committees may smooth that transition and, 
thus, can be regarded an indicator of impact. Taking into 
account the multi-stakeholder nature of public policy- 
making in democratic societies, this indicator includes not 
only public policy/ advisory committees of government 
agencies, but also those of supranational organisations, 
professional bodies, charities, and other stakeholders 
involved in public policy-making. 

In Oxford, 42% of respondents to the impact assess- 
ment survey reported participating on public policy/ 
advisory committees of various UK, overseas, and supra- 
national organisations, including the World Health 
Organization, the European Space Agency, UK Depart- 
ment of Health and other government agencies, UK par- 
liamentary committees, political parties, professional 
associations, medical charities, universities, and schools. 
The significance of these roles ranged from elected or 
appointed senior leadership positions to ordinary com- 
mittee membership or ad hoc advisory roles. Import- 
antly, several respondents suggested that participation 
on the advisory committees of charities should be 
included in impact assessment, although they had reser- 
vations about the additional burden of work that keeping 
full, accurate records of all these activities would require. 
One respondent noted: 

"I sit as a trustee of probably up to 12 charities, most 
of which have something to do with medical research. I 
think that sort of contribution is at least as important as 
contributions made to government activities. [However] 
I think it would be intolerable to have to keep a detailed 
account of all such activities and how would they be 
ranked relative to each other." 

Influence on public policy debate 

UK scientists see the news media as having a significant 
impact on public debate and many regard speaking to 
the media as the most effective method of communicat- 
ing with the public and influencing policy- makers [118]. 
Moreover, a survey of interactions with the mass media 
among researchers in France, Germany, Japan, the UK, 
and the US concluded that "the scientists most involved in 
these interactions tend to be scientifically productive, have 
leadership roles, and. . . that they perceive the interactions 
to have more positive than negative outcomes" [119]. 
Nonetheless, there may be a bias in reporting on the side 
of both the mass media and research institutions. Media 



interest in research may be linked to profound concerns 
about the evidence base for public policy, or with genuine 
interest in new knowledge, but it also feeds on what a re- 
spondent to a recent study called the "scientific entertain- 
ment value" of research [112]. A UK study suggested that 
"newspapers underreported randomised trials, emphasised 
bad news from observational studies, and ignored research 
from developing countries" [120]. A US study showed that 
news stories about new medical interventions are often in- 
accurate, imbalanced, or incomplete, and that as a result 
"may have a profound— and perhaps harmful— impact on 
health care consumers" [121]. But it is not only journalists 
who may be over- enthusiastic about weak science: press- 
releases by US academic medical centres themselves may 
have "a tendency to overstate the importance and down- 
play (or ignore) the limitations of research" [122]. For this 
reason, any media impact indicators should take into ac- 
count not only the number of citations and appearances, 
but also their quality, and should be used with caution. 

In Oxford, 42% of respondents to the impact assessment 
survey reported appearing in the local, national, or inter- 
national mass media during the five-year census period. 
Although no direct comparison is possible, this may be 
lower than implied by an earlier survey of epidemiology 
and stem cell researchers in the top five R&D countries 
for these fields, of whom 69% had been involved in inter- 
actions with the mass media in the last three years [119]. 
The range and magnitude of media impact in the context 
of this study can be assessed using data from the univer- 
sity's Press & Information Office. For example, Table 3 
shows the coverage of scientists from clinical medicine 
departments in conventional media in 2009. In addition, 
since October 2008, university clinical scientists have been 
able to disseminate their research through the iTunes U 
website, which in 2009 carried 6 free-to-download pod- 
casts relevant to clinical medicine: Cancer, Cancer in the 
Developing World, Childhood Diseases, Clinical Trials in 
Resource-Limited Settings, Pharmaceutical Industry, and 
Vaccine Research [123]. The number of downloads for 
such podcasts can be obtained from Apple Inc. and com- 
pared with other universities. 

Table 3 Media coverage of scientists from clinical 



medicine departments, 2009 



Type of media 


n 


Stories 

% 


UK & international press 


390 


62.3% 


UK & international online news* 


123 


19.6% 


UK radio 


60 


9.6% 


UK television 


53 


8.5% 


Total 


626 


100.0% 



^Unknown whether a story appeared in hard copy as well. 
Source: Press & Information Office, University of Oxford. 
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At the same time, it is important to note that media 
coverage does not necessarily lead to influence on public 
policy debate. Only 12% of respondents to the impact 
assessment survey reported perceptions of influencing 
public policy debate. Similar to some of the previous 
indicators, influence on public policy is a subjective indi- 
cator, and media coverage is only a weak measure of it. 
An important aspect of influencing public debates can- 
not be captured by such indicators: the potentially trans- 
formative influence of research on public discourses 
about policy and clinical practice includes shaping the 
ways in which such discourses frame problems and their 
solutions, and slowly changing the language in which 
they describe the social and physical worlds. 

Improved patient care or health outcomes 
Research income from the NHS and medical research 
charities 

Competitively- awarded NHS and charity funding is pre- 
dicated on the high quality of research and benefits for 
patients. The NHS Constitution puts research at its core 
because "[Research enables the NHS to improve the 
current and future health of the people it serves" [124]. 
Likewise, medical research charities aim to fund research 
that is important for the advancement of medical care 
[125]. Until the creation of the NIHR in 2006, NHS 
R&D funding was allocated to sustain the level of activ- 
ity in large teaching hospitals and it is argued to have 
been "far more a product of history, politics and prag- 
matic judgments than any rational analysis" [126]. Fol- 
lowing the creation of the NIHR, the majority of NHS 
R&D funding is now awarded competitively to NHS/uni- 
versity partnerships on the basis of peer review and 
bibliometric indicators [81]. Medical research charities 
have long been using peer review and research output 
indicators to allocate their research funding. In the run- 
up to the previous Research Assessment Exercises they 
urged government, the NHS and universities "to favour 
peer-reviewed charity research above funding that is not 
awarded competitively" [125]. Crucially, medical re- 
search charities aim to allocate research funding in order 
to achieve the greatest impact, and in fact they cham- 
pioned the idea of assessing the long-term benefits from 
health research in the UK [25]. Yet, there is evidence 
that competitively-awarded NHS or charity research 
funding does not necessarily achieve impact. An exten- 
sive assessment of the impact of the NHS Health Tech- 
nology Assessment Programme found that some of its 
projects achieved virtually no societal or economic im- 
pact [127]. It remains to be seen what proportion of 
competitively-awarded research funding predicated on 
impact actually achieves impact and which NHS bodies 
or medical research charities are most successful in 
funding research that achieves greater impact. 



Although the validity and reliability of using compe- 
titively-awarded research funding as a proxy indicator of 
impact requires further investigation, our study suggests 
that it is feasible to collect the required data and that 
this can be done with minimal burden on academic staff. 
HEFCE has recently recognised the competitive nature of 
NHS research funding, and to this effect the Higher Edu- 
cation Statistics Agency (HESA) is tasked with reporting it 
specifically alongside other competitively-awarded funding 
[34]. Charity funding is the most important source of ex- 
ternal research income in UK HEIs and its overwhelming 
majority is awarded competitively. In 2008/09, charity 
funding accounted for 51.2% and 44.2% of clinical medi- 
cine external research income in Oxford and all UK HEIs 
combined respectively (Table 2). Since the introduction of 
competitive mechanisms to allocate NHS R&D funding, 
NHS/university partnerships in Oxford have been success- 
ful in obtaining major awards, such as a NIHR Com- 
prehensive Biomedical Research Centre and NIHR Bio- 
medical Research Unit. But if we look at the percentage of 
research income from "UK central government bodies, 
local authorities, and health and hospital authorities" in 
Table 2, which mainly reflects research income from the 
NHS, it is still almost twice as low in Oxford (11.3%) as in 
all UK HEIs combined (20.9%). 

Measures of improved health services 

In the UK, up to 5%-10% of the medical workforce is 
made up of clinical academics - university employees 
who, in addition to their teaching and research duties, 
have honorary contracts with the NHS as practicing doc- 
tors [128]. As of 2009, there were over 125 full time 
equivalent (FTE) university- employed clinical academics 
in Oxford, who represented 4% of the clinical academic 
workforce in the UK [128]. They play a leading role in 
translating research from bench to bedside and make a 
significant contribution to the delivery and improvement 
of health services, especially those that are highly specia- 
lised and technology-intensive. Respondents to the im- 
pact assessment survey stressed the link between 
research and their own professional development, enab- 
ling state-of-the-art care for patients, which would 
otherwise take years to be established through changes 
to clinical guidelines. As we argued elsewhere, a strong 
link between practice, as praxis, and research involves 
their "organic" synergy in practitioners' everyday profes- 
sional lives, "which is more than having effective traffic 
of information between two otherwise discrete commu- 
nities and activities" [129]. 

Importantly, a number of respondents commented on 
impact assessment as an opportunity for better recogni- 
tion of the clinical aspects of their work. They argued 
that the conventional indicators of research output dis- 
advantaged clinical academics, compared to full-time 
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university scientists, who were described as having more 
time for basic research and better chances to publish it 
in journals with high impact factors. An anaesthetist, 
who was dissatisfied that in the previous RAE his/her 
specialty was judged solely on the basis of research out- 
puts, commented: 

"If these [impact] measures are introduced. . . it will be- 
come impossible for either universities locally or HEFCE 
nationally to ignore the clinical contributions of anaesthe- 
tists and focus narrowly upon research outputs." 

Overall, 13% of respondents reported improving health 
services, mainly in their local area. The range of impacts 
reported included: improving diagnostics and drug re- 
sponse prediction, fulfilling previously unmet clinical 
needs, making highly-specialised services more access- 
ible for the local communities, applying new recovery 
and rehabilitation strategies, and reducing waiting times 
and treatment costs. Given the wide range of impacts 
reported, it does not seem feasible to have one standar- 
dised measure of improved health services, and so, like 
in other practice-oriented fields, case studies can be a 
useful method to capture such impacts [130]. Nonethe- 
less, some standardised measures could be used as a 
proxy for the impact of universities on the delivery of 
health services, e.g. the number of university- employed 
clinical academics, their time (programmed activities) 
devoted to clinical work, and their translational research 
leadership and esteem as measured by NHS Clinical Ex- 
cellence Awards and NIHR Senior Investigator Awards. 

Changes to clinical or healthcare training, practice or 
guidelines 

Even though individual clinicians can adopt research find- 
ings directly from academic publications, mass adoption at 
a national or international level is usually facilitated by clin- 
ical guidelines and other evidence-based recommendations. 
Clinical guidelines, such as those produced by the National 
Institute for Health and Clinical Excellence (NICE), aim to 
improve the quality of healthcare in the UK by providing 
clinical recommendations based on the best available evi- 
dence [131]. Consequently, references to research in clinical 
and practice guidelines can be used as indicators of the 
likely application or adoption of research findings in clinical 
practice [132]. Although a citation in clinical guidelines 
does not guarantee an impact on health, it demonstrates a 
peer perception of research utility and can be considered as 
an intermediate outcome [26]. Research by the Wellcome 
Trust, the Medical Research Council (MRC), and the NIHR 
demonstrated the feasibility of employing conventional 
bibliometric analysis of research papers cited on clinical 
guidelines to assess the impact of biomedical research on 
healthcare policy and practice [24,26,133]. 

In Oxford, 20% of respondents to the impact assess- 
ment survey reported that their work had resulted in 



changes to clinical or healthcare training, practice or 
guidelines at a local, national, European, or international 
level. These reports were supported with evidence both 
of output (references in published guidelines) and of 
process (direct participation in the production of guide- 
lines). The majority of respondents mentioned that their 
research had been cited by, or that they participated in 
the working groups or committees of, the NICE, UK 
Royal Colleges, international professional associations, 
the World Health Organization, and other relevant bod- 
ies. A number of respondents also mentioned that their 
research was cited in NHS National Service Frameworks 
and Strategies. Inclusion in use-oriented systematic 
reviews, such as Cochrane Reviews, which screen studies 
for quality and relevance, was mentioned as another 
possible measure of impact. Whilst clinical guidelines, 
official recommendations, and systematic reviews lend 
themselves well to bibliometric analysis, some other 
types of evidence mentioned by respondents (e.g. leading 
on a new MSc course for practising physicians, develop- 
ing a section of the NHS Evidence web service, or pro- 
posing a new classification system for an infectious 
tropical disease) favour case studies. A case study 
method may be appropriate for claims about impacts on 
training and professional standards, accreditation frame- 
works, curriculum and teaching materials and textbooks, 
and pedagogical approaches in medical education. For 
these reasons, a mixed method approach, encompassing 
both bibliometric analysis and case studies, can be used 
to assess this area of impact. 

Development of new or improved drugs, treatments or 
other medical interventions; numbers of advanced phase 
clinical trials 

The clinical trial in human beings is a valid and apt indi- 
cator for the development of medical interventions be- 
cause currently it is "the preferred method in the 
evaluation of medical interventions" [134], representing 
"a key research activity with the potential to improve the 
quality of health care and control costs through careful 
comparison of alternative treatments" [135]. Moreover, 
experimental animal models and other pre-clinical stud- 
ies can be assessed on the basis of whether they have 
progressed to clinical trials, and the latter can be 
assessed on the basis of the benefits associated with each 
phase of clinical trials. Even though some medical inter- 
ventions can be difficult to classify, clinical trials are 
conventionally divided into phases I-IV. Phase I trials are 
generally designed as pharmacology studies; phase II trials 
are therapeutic exploratory investigations; phase III trials 
are assessments of the effectiveness of the new interven- 
tion; and phase IV trials are investigations into uncommon 
adverse effects of the new intervention [134]. Given the 
strategic intention of the NIHR to increase the number of 
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people in the UK participating in clinical trials and the 
health benefits of participating [30], the number of partici- 
pants in clinical trials is also an important indicator for 
the development of medical interventions. 

The University of Oxford runs one of the UK's largest 
clinical trials programmes in collaboration with many 
academic health centres in the UK and overseas. There 
are eight clinical trials units that have been awarded UK 
Clinical Research Collaboration (UKCRC) Registration, 
which recognises their capability to coordinate multi- 
centre clinical trials to a required standard [136]. During 
the census period, the university sponsored or partici- 
pated as a research site in 140 clinical trials involving 
154,888 participants (Table 4). According to the impact 
assessment survey, 32% of respondents reported contrib- 
uting to the development of new or improved drugs, 
treatments or other medical interventions. As one re- 
spondent noted, however, it is important to give appro- 
priate credit to early-phase translational research and 
basic science, which underpin advanced phase clinical 
trials with the most immediate patient benefits: 

"impact assessment needs to consider carefully the 
various stages of translational research so as to award 
credit correctly to those who have devised and brought 
to clinical evaluation new interventions, rather than giv- 
ing disproportionate credit to those who undertake late 
stage evaluation of technologies invented by others." 

Changes to public behaviour 

There is a growing recognition in medicine and public 
health that many diseases can be prevented or have their 



Table 4 University of Oxford sponsored clinical trials 
active, 2005/09 



Topic area 


Number 
of trials 


Phase 








Number of 
participants 




1 II 


III 


IV 


N/A 


Cancer 


7 


1 


4 


1 


1 


27314* 


Cardiac/circulation 


8 


1 


4 


3 




56335 


Diabetes 


7 






3 


4 


16350* 


Infectious disease 


25 


19 3 




1 


1 


1737* 


Neo/perinatal 


4 




1 


2 


1 


7220 


Nervous system 


2 




2 






312 


Obstetrics/gynaecology 


2 




1 


1 




240 


Paediatrics 


17 


8 


2 


6 


1 


5510 


Public health/primary care 1 1 180 


Psychiatry 


8 


1 2 




4 


1 


1853 


Renal 


2 




1 


1 




848 


Respiratory medicine 


4 


1 


1 


2 




686 


Tropical medicines 


53 


9 15 


8 


7 


14 


36303* 


Total 


140 


29 31 


24 


32 


23 


154888 



incomplete data. 

Source: Clinical Trials and Research Governance Team, University of Oxford. 



effects lessened by making and maintaining changes to 
public behaviour, and that demands and responsibilities for 
such changes should be placed not only on the individual, 
but also on the health system, the community, and the so- 
cial and political context [137]. It is a valid expectation on 
the part of research funders that the medical research com- 
munity should promote health-enhancing changes to pub- 
lic behaviour. At the same time, the measurement of health 
behaviours is challenging [137], and so is the attribution of 
behavioural change to specific research findings [25]. 

In Oxford, 7% of respondents to the impact assess- 
ment survey reported, or expressed hope, that their work 
had resulted in health- enhancing changes to public be- 
haviour. For example, one respondent assessed that 
since the implementation of the National Stroke Strategy 
and the establishment of a new academically-led Stroke 
Unit in the Oxford John Radcliffe Hospital: 

"[stroke] has become an emergency treatment, patients 
are arriving much more quickly and the public attitude 
towards stroke has changed insofar as patients now 
recognize it as a medical emergency." 

Another respondent cited an increased referral rate 
from UK regions and Ireland to demonstrate a greater 
awareness of a new life- extending treatment: 

"by initiating the FOXFIRE national clinical trial, 
patients and relatives are seeking the availability of radio- 
embolisation treatment at centres such as Oxford." 

Yet others cited activities, which they hoped would 
change health-risk behaviours: 

• "I would hope that our work on the adverse effects 
of obesity on heart and arteries has led to increased 
awareness of obesity as a risk factor" 

• "[I was a] Member of Tobacco Control Forum that 
advised on legislation on smoking ban in restaurants 
and public houses" 

• "I have participated in the Cancer Research UK 
SunSMART campaign that aims to increase public 
awareness of exposure to sunlight and the risks of 
developing skin cancer." 

Given that it is extremely challenging to compare such 
diverse behavioural changes and verify to what extent 
they are attributable to specific research, case studies, 
complemented by standardised measures of health out- 
comes, can be used to describe such changes, explore 
their connections with underpinning research, and esti- 
mate their potential health benefits. For more robust 
conclusions, however, more specialised and adequately 
resourced impact evaluation studies would be required. 

Measures of improved health outcomes 

Research can lead to improved health outcomes through 
many different pathways, including improvements in 
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health services, changes to clinical guidelines and train- 
ing, development of new drugs and treatments, and 
changes to public behaviour as discussed above, but also 
through the development of new methods to measure 
health status [6]. Despite the multitude of pathways and 
disease states, there are standardised measures for asses- 
sing health outcomes, such as Quality Adjusted Life 
Years (QALYs), Disability Adjusted Life Years (DALYs), 
Health Adjusted Life Expectancy (HALE), and Medical 
Outcomes Study 36-item Short-Form Health Survey 
(SF-36) that can be used in research impact assess- 
ment [6,18,25]. The key advantage of such measures 
is that "they serve as a common metric to allow hin- 
ders to assess the value of an investment across dis- 
ease states" [18]. 

In Oxford, 17% of respondents to the impact assess- 
ment survey reported that their work had directly or in- 
directly improved health outcomes in the UK and/or 
overseas. The range of suggested improvements included 
reduction in mortality, morbidity, and accumulated dis- 
ability; improvements in the quality of life of patients 
and their carers; as well as the development of new 
disease-specific patient reported outcome measures 
(PROMs). For all that, none of the respondents were 
able to provide quantified outcome measures and many 
expressed concerns that it was difficult to attribute, ver- 
ify, quantify, and meaningfully compare improvements 
in health outcomes across various diseases and popula- 
tions. For example, how does clinical islet transplant- 
ation leading to insulin-independency and reversal of 
life-threatening hypoglycaemic unawareness in UK 
patients, cited by one respondent, compare to research 
on the national introduction of the pneumococcal vac- 
cine in the Gambia, cited by another respondent? These 
responses suggest that there is a need for research hin- 
ders and the medical research community to develop 
uniform guidelines for the identification of the popula- 
tions who can benefit from various interventions, the 
level of benefit, and the extent of implementation. Pro- 
vided such guidelines are developed, the standardised 
measures of health outcomes such as QALYs, DALYs, or 
HALE can be employed by universities to make initial 
estimates of health benefits, which may be subsequently 
validated by research funders and used in conjunction 
with case studies. 

Cultural enrichment, including improved public engagement 
with science and research 

Increased levels of public engagement with science 
and research 

Public engagement, or science communication and pub- 
lic understanding of science, as it is also known, is an in- 
clusive term that refers to "the many ways in which 
higher education institutions and their staff and students 



can connect and share their work with the public." 
[138]. The Concordat for Engaging the Public with Re- 
search, which is signed and supported by major UK hin- 
ders and stakeholders, recognises "the importance of 
public engagement to help maximise the social and eco- 
nomic impact of UK research" and suggests that "UK re- 
search organisations have a strategic commitment to 
public engagement" [139]. Likewise, the NIHR has a 
strategic commitment to involving patients and the pub- 
lic in all stages of NHS R&D, because it sees such en- 
gagement as leading "to research that is more relevant 
to peoples needs and concerns, more reliable and more 
likely to be put into practice" [30]. For many years, vari- 
ous research funding organisations, including charities, 
have been running schemes of additional funding for 
public engagement and outreach projects; the amount 
and number of such awards may be a proxy for univer- 
sities' interest in public engagement activity. 

At the same time, a study by the Royal Society found 
that, in general, scientists did not prioritise public en- 
gagement activities because of the need to spend more 
time on research [140]. They did, however, express will- 
ingness to engage more with the public and believed that 
including public engagement in the Research Assess- 
ment Exercise would provide an incentive to do so 
[140]. While there is a choice of possible indicators of 
institutional interest and investment in public engage- 
ment and outreach activity, and adequate measures and 
proxies can be developed, they are of limited use in cap- 
turing the outcomes of such activities and their actual 
benefits for the populations concerned [112]. These lim- 
itations were powerfully stated in the report of a recent 
study of 15 European countries, which showed that 
"analysis and evaluation of PE [public engagement] ac- 
tivities is still underdeveloped, lacking robust and shared 
indicators of output and performance" [141]. Whereas 
standardised surveys can be used successfully to collect 
data about public engagement activity [118,140,141], 
there remains a challenge for HEFCE to develop rigor- 
ous methods to assess their quality and outcomes - both 
for the purpose of the REF and to support those univer- 
sities who wish to promote it on the institutional level. 

In Oxford, 54% of respondents to the impact assess- 
ment survey reported participating in patient and public 
involvement activities during the five-year census period. 
Although no direct comparison is possible, this may be 
lower than implied by national, multi-disciplinary stud- 
ies. For example, the 2006 Royal Society survey found 
that 74% of scientists and engineers in various disci- 
plines, including clinical medicine, had participated in 
science communication or public engagement activities 
in the year prior to the survey date [140], while a 2000 
survey by the Wellcome Trust and MORI found that 
52% of clinical biomedical scientists had participated in 
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such activities in the previous year [118]. The patient 
and public involvement activities reported in this survey 
were very diverse in nature, including acting as a Science 
and Engineering Ambassador, giving public lectures and 
talks to patient groups, communicating with patients 
and clinical trials volunteers, acting as a patron to a 
patient support group, participating in the NIHR Bio- 
medical Research Centre (BRC) Open Days, collabora- 
ting with a Wellcome Trust Artist-in-Residence, being 
interviewed by a journalist, maintaining a public web- 
site, engaging with policy-makers and NGOs, working 
with science centres and museums, organising work- 
shops for pupils at local schools, and judging various 
competitions. 

Given the many ways in which public engagement ac- 
tivity manifests itself, it is challenging to compare them 
in a standardised manner. For example, one respondent 
mentioned "public events to raise awareness of issues 
surrounding stem cell biology: World Economic Forum 
Annual Meeting, Davos. . ., [local schools], Witney." 
How does researchers' engagement with global leaders 
in Davos compare with their engagement with school- 
children in the local community, and how does one esti- 
mate the value to the local community of world research 
leaders engaging with schoolchildren? Moreover, because 
the majority of respondents participated in several 
events every year the burden of collecting data about 
public engagement activity may be substantial. As one 
respondent - whose involvement with the British Sci- 
ence Association over several decades had led to many 
developments, including the creation of a science com- 
munication award - remarked: 

"It is really too much to expect that one should keep a 
detailed account of all such activity and would indeed be 
inhibiting of such activity." 

In order to minimise the burden and standardise indi- 
cators, HEFCE can draw on the Wellcome Trust and 
Royal Society studies to develop a discipline-specific 
public engagement survey, which can be administered as 
part of the annual HEFCE Higher education-business 
and community interaction survey (HE-BCI). 

Improved social welfare, social cohesion or national 
security 

Measures of improved social equity, inclusion or cohesion 

There are several important ways in which universities 
can improve social equity, inclusion, and cohesion. Uni- 
versities can enhance diversity and equality among their 
students and faculty. The percentage of women and eth- 
nic minorities among applicants and matriculants to 
medical schools are duly regarded as key indicators in 
academic medicine [142,143]. In the UK specifically, the 
NIHR requires that universities applying for the next 
round of translational research funding hold at least the 



silver award of the Athena SWAN Charter for women 
in science [144]. Through a competitive application pro- 
cess, Athena SWAN confers bronze, silver and gold 
awards, which "recognise and celebrate good practice in 
recruiting, retaining and promoting women in STEMM 
[Science, Technology, Engineering, Medicine and Math- 
ematics] in higher education" [145]. Nevertheless, such 
indicators of diversity, equality, and good practice are 
not regarded as impact indicators by HEFCE and thus 
were not included in the 2010 impact pilot. Tradition- 
ally, inter alia, such indicators are included in the "En- 
vironment" section of the RAE/REF, which will carry a 
weighting of 15% in the overall outcome of the REF. In 
countries with privately funded health care systems, 
medical schools and academic medical centres play an 
important role in improving social equity, inclusion, and 
cohesion by providing uncompensated health services to 
the populations most at risk of being underserved, i.e. 
the uninsured and members of disadvantaged communi- 
ties [146]. Measuring the extent of such health services, 
both in terms of their monetary value and percentage in 
the overall provision of health services, would be an im- 
portant indicator of societal and economic impact. In 
the UK, however, everyone has equal access to academic 
medicine because the National Health Service is publicly 
funded, and it is free at the point of use for everyone 
who is resident in the UK. Yet, there is some variation 
in the morbidity and mortality rates as well as in the 
provision and quality of services across the UK, which 
medical schools and university-employed clinical aca- 
demics working in the NHS can address. For example, 
Oxford clinical academics contributed to the design and 
implementation of the National Stroke Strategy to re- 
duce regional variations in the quality of stroke care and 
to the establishment of a new academically-led Stroke 
Unit in the Oxford John Radcliffe Hospital to improve 
stroke care in Oxford and Oxfordshire. Such contribu- 
tions can be captured through the measures of improved 
health services and changes to public policy, as discussed 
above. 

Whereas health is a universal human right [147], dis- 
advantaged populations in less developed countries are 
trapped in the vicious circle of poverty, lack of educa- 
tion, social inequalities, tropical diseases, and poor 
health [148,149]. The responsibility of the international 
community to uphold human dignity, equality and 
equity is encompassed in the eight United Nations Mil- 
lennium Development Goals (MDGs), each of which has 
its own targets, with measurable indicators [150]. 
Through its initiative "Academic Impact", the United 
Nations seeks a commitment on the part of research- 
intensive institutions of higher education to "the funda- 
mental precepts driving the United Nations mandate, in 
particular the realization of the universally determined 
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Millennium Development Goals" [151]. There are four 
MDGs to which medical research can contribute by fo- 
cusing on the health needs of disadvantaged populations 
in less developed countries: 

• reduce child mortality; 

• improve maternal health; 

• combat HIV/ AIDS, malaria, and other diseases; and 

• develop a global partnership for development. 

Research in tropical medicine and global health is the 
major avenue for clinical scientists in Oxford to contrib- 
ute to the achievement of the MDGs. The university has 
tropical medicine research groups, who are permanently 
based in Wellcome Trust- Oxford University centres in 
Kenya, Thailand, Viet Nam, Laos, Tanzania, Indonesia 
and Nepal, as well as collaborators around the world 
[152]. Their activities range from basic, epidemiological 
and clinical research to behavioural sciences and use of 
public health evidence to monitor progress towards the 
MDGs [152]. Crucially, capacity building is integral to 
all tropical medicine and global health activities, and 
several respondents to the impact assessment survey 
emphasised that capacity building in developing coun- 
tries should be included in impact indicators. As an ex- 
ample of their impact in sub-Saharan Africa, several 
respondents mentioned setting up clinical trials, purchas- 
ing high-specification equipment, large-scale long-term 
AIDS education and prevention programmes, training pro- 
grammes for practitioners and researchers, and fundraising 
activities. Such activities provided unique resources to the 
region concerned and contributed to the development of 
specialised skills, as well as having had an educational 
impact on important segments of the population. It fol- 
lows that in addition to the targets and measurable indi- 
cators encompassed in the MDGs, impact on health 
equality needs to include case studies focusing on 
research-based capacity building activities, such as men- 
toring the local academic and clinical workforce, develop- 
ing local infrastructure, and contributing to education in 
local communities. 

Application of new security technologies or practices 

The UK National Security Strategy recognises that in 
todays globalised world infectious diseases are among 
the major security challenges, and that the highest risk is 
an influenza- type pandemic [153]. It is estimated by the 
government that there is a high probability of such a 
pandemic occurring and that "possible impacts of a fu- 
ture pandemic could be that up to one half of the UK 
population becomes infected, resulting in between 
50,000 and 750,000 deaths in the UK, with correspond- 
ing disruption to everyday life" [153]. In view of this, 
research-intensive universities and academic health 



centres have an important role in improving national 
and global security by addressing the risks of infectious 
diseases. In Oxford, examples of research-based activities 
aimed at addressing the risk of infectious diseases in- 
clude the development and patenting of new technolo- 
gies that subsequently were used by pharmaceutical 
companies for the preparation of H1N1 (swine flu) and 
H5N1 (bird flu) vaccines. In another example, following 
the declaration of a global H1N1 (swine flu) pandemic 
by the World Health Organization in June 2009, clinical 
scientists in Oxford, together with their colleagues in 
Bristol, Exeter, London, and Southampton, conducted 
the first trials of H1N1 vaccines in the UK and provided 
important information to the Department of Health to 
guide immunisation policy in the UK [154,155]. In order 
to assess the impact of such activities, both case studies 
and standardised measures of potential health outcomes 
can be used. 

Limitations 

This article presents, to our best knowledge for the 
first time, empirical data to demonstrate how impact 
assessment operates and could operate in academic 
clinical medicine, providing funders and the medical 
research community with potential benchmarks for fu- 
ture comparative studies. However, our study has sev- 
eral methodological and data limitations, which entail 
the possibility of bias in the results. Administrative 
data for the proposed impact indicators were gathered 
retrospectively. If the university and faculty members 
were asked in advance to collect the types of data 
required for impact assessment, the scope and granu- 
larity of the data presented here are likely to have 
been greater. Moreover, there are limitations on gener- 
alising from a voluntary survey of a relatively small 
population. Although the achieved response rate of 
48.1% was relatively good, and sample proportions by 
gender, career stage, and staff category were similar to 
the population surveyed, there were slight discrepan- 
cies between our sample and the population surveyed 
(Table 1). Therefore, we cannot rule out the possibility of 
self-selection bias. Finally, a compulsory census of the 
entire population of clinical academics and scientists 
working in the university and its partner hospitals would 
have probably yielded different results. Given that univer- 
sities can select which and how many faculty members to 
include in their institutional submissions to the RAE/ 
REF [156], not all eligible faculty members are usually 
submitted because universities try to maximise their 
funding by selecting more accomplished faculty mem- 
bers. The grounds for selection may differ between those 
faculty members whose individual publications are 
included in the research outputs submitted by the unit, 
and those whose work is referenced in the impact part of 
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the submission [157]. For the purpose of the REF impact 
pilot exercise, we surveyed only those faculty members 
whose publication outputs were submitted to the 2008 
RAE and thus were most likely to have their publications 
submitted to the 2014 REF. As demonstrated in Table 1, 
the survey population is skewed towards senior and mid- 
career university scientists. If we surveyed the entire 
population of clinical academics and scientists working 
in the university and its partner hospitals, we would have 
probably found more examples of impact, especially from 
clinical practice, which would have most likely been 
spread more evenly across the population. 

Implications for performance-based research funding 
systems 

Notwithstanding methodological and data limitations, 
the results of our study demonstrate the range and sig- 
nificance of impacts in clinical medicine in a given uni- 
versity, and enable us to critically assess the "common 
menu" of impact indicators proposed by HEFCE in 
2010. While the majority of the proposed indicators have 
some validity, there are significant challenges in refining 
the current indicators and methodologies in a number 
of ways. 

First, the wide range and significance of impacts cap- 
tured in our study suggest that a clearer conceptualisa- 
tion and standardisation of impact measures is required 
in order to increase the validity and reliability of impact 
assessment. The issue of validity relates to the degree of 
certainty that the proposed indicators measure what they 
claim to measure, i.e. research impact. For example, it is 
not certain that participation on public policy commit- 
tees or changes to legislation reflect ones research activ- 
ity rather than a sense of civic duty. The issue of 
reliability is concerned with the consistency of measure- 
ments. Without applying precise measures of impact 
and criteria for the attribution of impact to specific re- 
search activity across all universities, any impact assess- 
ment will be inconsistent and, thus, unreliable. While 
developing new standardised measures can be very 
costly and challenging, this can be mitigated by concen- 
trating on those indicators that are already standardised, 
e.g. research income, patents, and clinical trials. The 
danger of doing so, however, is that such indicators have 
not been specifically designed for measuring impact and 
thus can only be used as proxy indicators, and that the 
probabilities of impact associated with such proxy indica- 
tors are currently unknown. As discussed in the case of re- 
search funding, although research funding competitively- 
awarded for the purpose of improved patient care is much 
more likely to lead to improved patient care in a relatively 
short period of time than non-competitively-awarded 
funding for basic science research, the probabilities of 
achieving impact within specified time horizons and how 



such probabilities vary between different medical research 
funders are currently unknown. Moreover, it should be 
kept in mind that an obvious limitation of any attempt to 
develop and use either direct or proxy impact indicators 
for high-stakes assessments of research is that their ro- 
bustness as indicators is threatened once they become tar- 
gets and benchmarks for performance (Goodhart's law). 

Second, the aggregation of impacts is not yet possible 
because the majority of the indicators studied are not 
standardised. While the data collected using the current 
indicators provide evidence of the wide range and sig- 
nificance of impacts and can be used for descriptive case 
studies, we are not able to add the data collected from 
individual researchers and teams together to create an 
aggregate indicator for all faculty members in a given 
university and then compare it across universities. If im- 
pact assessment is to provide incentives for every faculty 
member to increase the economic and social impact of 
their research, then the impacts of the research of every 
faculty member must be counted in. Likewise, if impact 
assessment is to reward those universities that deliver 
more benefits to the economy and society, then we need 
to find a way to aggregate various impact indicators into 
the net indicator of impact in order to compare univer- 
sities. The trusty method of traditional peer review will 
be stretched to the limit by the demand to weigh all the 
different types of evidence for each of the indicators, as- 
sess their quality, and produce consistent, meaningful 
aggregate profiles for each submitting institution. 

Third, there is a challenge in choosing between 
quantitative and qualitative methodologies. Our study 
demonstrates that some of the current indicators lend 
themselves well to being quantified, but some others 
can be described only qualitatively. The advantage of 
using quantitative indicators is that they can be stan- 
dardised and aggregated, allowing universities to use 
them on a continuous basis to track their impact, com- 
pare it with other universities, and recognise the contri- 
bution of every faculty member, of whatever scale. At 
the same time, developing valid and reliable quantita- 
tive indicators and then collecting data to operational- 
ise them may be costly and time intensive. The 
advantage of using qualitative indicators and case stud- 
ies is that there are important areas of impact that can 
be captured only qualitatively, and evaluating research 
impact through qualitative case studies is a relatively 
quick and cost-contained alternative to quantitative 
indicators. However, the emphasis on qualitative indi- 
cators would stretch traditional peer review further and 
concentrate on the most prominent examples of im- 
pact, overlooking more modest contributions. 

Fourth, our study highlights the necessity to balance 
carefully incentives for basic science, translational re- 
search, and clinical work across impact indicators and 
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between impact assessment and conventional research 
outputs because the rewards associated with impact as- 
sessment and even impact indicators themselves can 
modify the behaviour of individuals and universities. On 
the one hand, it takes considerably more time and effort 
to make basic science discoveries and design new inter- 
ventions than to evaluate and bring them to market. Ex- 
cessive emphasis on impact may give disproportionate 
rewards to those who evaluate interventions discovered 
by others and bring them to market, without giving full 
recognition to basic researchers, who invent and bring 
new interventions to clinical evaluation. On the other 
hand, bibliometrics and other conventional indicators of 
research output recognise the work of basic scientists 
who publish in journals with high impact factors and are 
part of large groups of authors, but may disadvantage 
academic physicians who devote a significant part of 
their time to clinical work and publish in specialty- 
specific journals. 

Fifth, there is a need for clear, collaboratively- 
developed guidelines for university-based information 
systems and data collection procedures. The current 
university information systems and procedures were not 
specifically designed to provide certain types and levels 
of granularity of data required for impact assessment, 
and individual researchers often do not keep a record of 
events and facts that can be interpreted as impacts. As a 
result, collecting impact data retrospectively puts a 
strain on the scope and accuracy of impact indicators 
and presents a significant burden upon universities and 
their faculty. At the same time, in addition to HEFCE, a 
number of publicly-funded research councils and med- 
ical research charities have started asking universities 
and individual grant recipients to provide evidence of 
the economic and social impact of sponsored research. 
For these reasons, collaboration between HEFCE and 
other public funding agencies, medical research char- 
ities, universities, and the wider medical research com- 
munity is required in order to develop and validate a 
common set of impact indicators with universal guide- 
lines on data collection. This will raise the quality of im- 
pact assessment, reduce the duplication of effort, 
minimise the burden of collecting data for different 
types of impact indicators, and allow faculty to engage 
with patient populations and local communities more 
effectively. 

Finally, our study suggests that there is a potential for 
impact indicators to be used for objectives other than 
the allocation of public research funds under the REF, 
i.e. medical schools can use impact indicators as part of 
their social accountability strategies. Although the allo- 
cation of UK government funding to universities on the 
basis of impact provides universities with incentives to 
increase their societal and economic impact, it is 



unlikely that these incentives alone can change the long- 
term behaviours of universities. On the one hand, these 
incentives may not be sufficiently high to change the 
long-term behaviour of universities in clinical medicine, 
because government funding is no longer the main 
source of research funding in clinical medicine and only 
a small proportion of such funding is proposed to be 
allocated on the basis of impact. It is also impossible to 
use legal mechanisms to enforce the accountability of 
universities and individual researchers for the receipt of 
research funding predicated on impact, because impact, 
as it is currently defined and measured, cannot be 
included in research contracts. Furthermore, universities 
are highly devolved institutions, to the effect that deans 
of medical schools or heads of academic departments 
have no direct hierarchical powers to hold individual 
researchers accountable for the actual impact of their re- 
search, which, according to the current definition, is 
expected to be achieved within ten years. On the other 
hand, there is growing consensus among medical schools 
around the world that they need to achieve and demon- 
strate a greater impact on the health needs of the popu- 
lations and societies they serve as part of their social 
accountability project. This goal is consistent with the 
goal of the government to increase the economic and so- 
cial impact of publicly-funded research through the REF. 
If universities, public funding agencies, medical research 
charities, and the wider medical research community 
collaboratively develop measurable impact indicators 
and methodologies that enable universities to claim, 
track and compare the impacts of their research in a 
transparent and rigorous manner on a continuous basis, 
universities can use impact assessment as part of their 
social accountability strategies. This way, the medical re- 
search community can adjust their long-term behaviour 
in line with the requirements of social accountability 
and, in doing so, once again prove to the public and tax- 
payers its right to professional autonomy and self- 
regulation. 

Conclusion 

Overall, the evidence from our study supports the claim 
that assessing impact is feasible, but that current meth- 
odologies will need to be significantly improved before 
using measurable impact indicators as a basis on which 
to change the long-term behaviour of universities. The 
impact indicators and methods discussed in this paper 
can be used successfully to identify the many areas 
where impacts (in the REF sense) occur and the wide 
range of forms that they take, in order to build an evi- 
dence base for descriptive case studies of impact and im- 
pact statements as part of the REF peer review in 2014. 
However, a quinquennial peer review of descriptive case 
studies and statements, and the funding incentives 
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associated with it, are unlikely to elicit and sustain 
change in the long-term behaviour of universities in the 
field of clinical medicine. To do so, universities would 
need to be able to claim, track and compare the impacts 
of their research in a transparent and rigorous manner 
on a continuous basis as part of their social accountabil- 
ity strategies. This is a different task, which would re- 
quire further debate about impact assessment to move 
away from a quinquennial peer review of descriptive 
evidence to a continuous monitoring and analysis of 
measurable impact indicators. The latter would entail 
developing a set of valid and reliable indicators; robust 
methodologies for attributing and aggregating impact; 
carefully balanced incentives for basic science, transla- 
tional research, and clinical work; and clear guidelines 
for universities. Given the scope and complexity of 
these tasks, universities and the wider medical research 
and academic community have an important role to play 
in shaping public understandings of research impacts, of 
the nature of evidence required to claim and assess 
impacts, and of the robustness of methodologies and 
indicators for impact assessment in the REF 2014 and 
beyond. 
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